Preoperative survival prediction method based on enhanced medical images and computing device using thereof

ABSTRACT

A preoperative survival prediction method and a computing device applying the method include constructing a data seta according to a plurality of enhanced medical images and a resection margin of each enhanced medical image and obtaining a plurality of training data sets from the constructed data set. For each training data set, multi-task prediction models are trained. A target multi-task prediction model is selected from the plurality, and a resection margin prediction value and a survival risk prediction value are obtained by predicting an enhanced medical image to be measured through the target multi-task prediction model. The multi-task prediction model more effectively captures the changes over time of the tumor in multiple stages, so as to enable a joint prediction of a resection margin prediction value and a survival risk prediction value.

FIELD

The present disclosure relates to a technical field of medicaltechnology, specifically a preoperative survival prediction method basedon enhanced medical images and a computing device using thereof.

BACKGROUND

Pancreatic ductal adenocarcinoma (PDAC) is a human cancer, which has anextremely very poor 5-year survival rate of 9%. Surgical sectioning, incombination with neoadjuvant chemotherapy, is the only potentiallycurative treatment for PDAC patients. However, outcomes varysignificantly even among the resected patients of the same stagereceiving similar treatments. Accurate preoperative prognosis of PDACfor personalized treatment is thus highly desired.

Previous work adopts image texture analysis for the prediction ofsurvival of PDAC. However, the representation power of hand-craftedfeatures only on the venous phase in computer tomography (CT) may belimited. More recently, cancer outcome prediction models based on deeplearning have shown good powers of prediction for lung cancer andgliomas. The success of 3DCNNs contributes not only to the capture ofdeep features in the 3D gross tumor volume but also in the peritumoralregions. However, such models may not generalize well for PDAC becauseimportant predictive information may not exist in the isolated imagingmodality/phase.

A solution for preoperative survival prediction is required.

SUMMARY

A first aspect of an embodiment of the present disclosure provides apreoperative survival prediction method based on enhanced medicalimages. The method includes: constructing a data seta according to aplurality of enhanced medical images and a resection margin of eachenhanced medical image, and obtaining a plurality of training data setsfrom the constructed data set. For each training data set, inputting thetraining data set into a first network structure and a second networkstructure for training, extracting first feature maps of the trainingdata sets through the first network structure, and extracting secondfeature maps of the training data sets through the second networkstructure. Obtaining joint feature maps by connecting the first featuremaps and the second feature maps, obtaining a resection margin risk lossvalue by calculating a resection margin risk loss function based on thejoint feature maps, and obtaining a survival risk loss value bycalculating a survival risk loss function based on the joint featuremaps Determining whether the resection margin risk loss value and thesurvival risk loss value meet their respective loss thresholds and whenthe resection margin risk loss value and the survival risk loss valueboth meet their respective loss thresholds, stopping the training of thefirst network structure and the second network structure to obtain aplurality of multi-task prediction models. Selecting a target multi-taskprediction model from the plurality of multi-task prediction models andobtaining a resection margin prediction value and a survival riskprediction value by predicting an enhanced medical image to be measuredthrough the target multi-task prediction model.

A second aspect of an embodiment of the present disclosure provides acomputing device, which includes at least one processor and a storagedevice storing one or more programs which when executed by the at leastone processor, causes the at least one processor to construct a dataseta according to a plurality of enhanced medical images and a resectionmargin of each enhanced medical image, and obtain a plurality oftraining data sets from the constructed data set. For each training dataset, input the training data set into a first network structure and asecond network structure for training, extract first feature maps of thetraining data sets through the first network structure, and extractsecond feature maps of the training data sets through the second networkstructure. Obtain joint feature maps by connecting the first featuremaps and the second feature maps, obtain a resection margin risk lossvalue by calculating a resection margin risk loss function based on thejoint feature maps, and obtain a survival risk loss value by calculatinga survival risk loss function based on the joint feature maps. Determinewhether the resection margin risk loss value and the survival risk lossvalue meet their respective loss thresholds and when the resectionmargin risk loss value and the survival risk loss value both meet theirrespective loss thresholds, stop the training of the first networkstructure and the second network structure to obtain a plurality ofmulti-task prediction models. Select a target multi-task predictionmodel from the plurality of multi-task prediction models and obtain aresection margin prediction value and a survival risk prediction valueby predicting an enhanced medical image to be measured through thetarget multi-task prediction model.

A third aspect of an embodiment of the present disclosure provides anon-transitory storage medium having stored thereon instructions that,when executed by a processor of a computing device, causes the computingdevice to perform a method for preoperative prediction of survival. Themethod includes constructing a data set according to a plurality ofenhanced medical images and a resection margin of each enhanced medicalimage, and obtaining a plurality of training data sets from theconstructed data set. For each training data set, inputting the trainingdata set into a first network structure and a second network structurefor training, extracting first feature maps of the training data setsthrough the first network structure, and extracting second feature mapsof the training data sets through the second network structure.Obtaining joint feature maps by connecting the first feature maps andthe second feature maps, and obtaining a resection margin risk lossvalue by calculating a resection margin risk loss function based on thejoint feature maps, and also obtaining a survival risk loss value bycalculating a survival risk loss function based on the joint featuremaps. Determining whether the resection margin risk loss value and thesurvival risk loss value meet their respective loss thresholds; when theresection margin risk loss value and the survival risk loss value bothmeet their respective loss thresholds, stop the training of the firstnetwork structure and the second network structure to obtain a pluralityof multi-task prediction models. Selecting a target multi-taskprediction model from the plurality of multi-task prediction models andobtaining a resection margin prediction value and a survival riskprediction value by predicting an enhanced medical image to be measuredthrough the target multi-task prediction model.

In the embodiments of the present disclosure, by constructing a dataseta according to a plurality of enhanced medical images and a resectionmargin of each enhanced medical image, and obtaining a plurality oftraining data sets from the constructed data set. For each training dataset, a multi-task prediction models is trained. A target multi-taskprediction model is selected from the plurality of multi-task predictionmodels and a resection margin prediction value and a survival riskprediction value are obtained by predicting an enhanced medical image tobe measured through the target multi-task prediction model. Themulti-task prediction model captures more effectively the time changesof the tumor in multiple stages, so as to make a joint prediction of aresection margin prediction value and a survival risk prediction value.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic flow chart of an embodiment of a preoperativesurvival prediction method based on enhanced medical images according tothe present disclosure.

FIG. 2 shows a schematic diagram of an example of the multi-phase(dynamic) CE-CT images and pancreatic ductal adenocarcinoma (PDAC) tumorenhancement patterns according to the present disclosure.

FIG. 3 shows a schematic structural diagram of a network architecture ofa multi-task prediction model according to the present disclosure.

FIG. 4 shows a schematic structural diagram of an embodiment of a riskstratification result according to the present disclosure.

FIG. 5 shows a schematic structural diagram of an embodiment of apreoperative survival prediction device based on enhanced medical imagesaccording to the present disclosure.

FIG. 6 shows a schematic structural diagram of a computing deviceapplying the method according to the present disclosure.

DETAILED DESCRIPTION

The embodiments of the present disclosure are described with referenceto the accompanying drawings. Described embodiments are merelyembodiments which are a part of the present disclosure, and do notinclude every embodiment. All other embodiments obtained by those ofordinary skill in the art based on the embodiments of the presentdisclosure without creative efforts are within the scope of the claims.

Terms such as “first”, “second” and the like in the specification and inthe claims of the present disclosure and the above drawings are used todistinguish different objects, and are not intended to describe aspecific order. Moreover, the term “include” and any variations of theterm are intended to indicate a non-exclusive inclusion. For example, aprocess, a method, a system, a product, or a device which includes aseries of steps or units is not limited to steps or units which arelisted, but can include steps or units which are not listed, or caninclude other steps or units inherent to such processes, methods,products, and equipment.

FIG. 1 shows a schematic flow chart of an embodiment of a preoperativesurvival prediction method based on enhanced medical images according tothe present disclosure.

As shown in FIG. 1 , the preoperative survival prediction method basedon enhanced medical images applicable in a computing device can includethe following steps. According to different requirements, the order ofthe steps in the flow chart may be changed, and some steps may beomitted.

In block 11, constructing a data set according to a plurality ofenhanced medical images and a resection margin of each enhanced medicalimage, and obtaining a plurality of training data sets from theconstructed data set.

In some embodiments, a series of medical images are obtained by scanninga patient's body or a part with an image scanning device afterintravenous injection of contrast media. Medical images obtained byscanning a patient using an image scanning device after intravenousinjection of contrast media are called enhanced medical images. Enhancedmedical image corresponding to different phases between a plurality ofplanes can be considered as one frame in the series of enhanced medicalimages. That is, the series of enhanced medical images will include aplurality of enhanced medical images. With application of contrast mediafor clinical purposes, effectiveness of images is enhanced, and accuracyof diagnoses is improved.

The patient may be a patient suffering from any tumor, such as apancreatic ductal adenocarcinoma, a liver tumor, a lung tumor, alymphoma tumor, a hepatic hemangioma, or the like. The image scanningdevice can utilize, for example, a Computer Tomography (CT), a MagneticResonance Imaging (MRI), a positron emission tomography (PET), a singlephoton emission computed tomography (SPECT), an ultrasound scanning, arotational angiography, and other medical imaging modalities.Correspondingly, the enhanced medical image can be an enhanced CT image,an enhanced MRI image, an enhanced PET image, an enhanced SPECT image,an enhanced ultrasound scan image, and an enhanced rotationalangiography image and other enhanced medical imaging modal images.

In order to facilitate understanding of the present disclosure, thefollowing describes an example of enhanced CT images of PDAC patients.

In some embodiments, the method of constructing a data set according toa plurality of enhanced medical images and a resection margin of eachenhanced medical image includes: obtaining a plurality of first targetimages by delineating a first target region in each enhanced medicalimage corresponding to a first phase; obtaining a plurality of secondtarget images by segmenting a second target region in each enhancedmedical image corresponding to a second phase; constructing an array bycombining one enhanced medical image and the corresponding first targetimage, the corresponding second target image, and the correspondingresection margin, the data set including a plurality of the arrays.

Phases shown in the enhanced CT images of the pancreatic ductaladenocarcinoma patient include a non-contrast phase, a pancreatic phase,and a venous phase. The non-contrast phase is before any contrast mediais injected into body of the patient. The pancreatic phase (also calledarterial phase) is when a contrast media is moved from the heart intothe arteries, all structures/organs that get their blood supply from thearteries will show optimal enhancement. In the aorta, a majorenhancement can be observed. The venous phase is when the contrast mediais in the veins, flowing back to the heart. In the venous phase, theliver parenchyma is enhanced through blood supply by the portal vein andsome enhancement of the hepatic veins can be seen. Pancreatic CT scansof 205 patients (with significant PDACs, meaning tumor size=2.5 cm) wereundertaken preoperatively during non-contrast, pancreatic, and portalvenous phases (i.e., 615 CT volumes). Only 24 out of 205 patients haveR1 resection margins, and the imbalance in a resection margin loss isconsidered.

In this embodiment, the first phase can be the pancreatic phase, and thesecond phase can be the venous phase. The first target region in eachenhanced medical image can be a pancreas region. The second targetregion in each enhanced medical image can be a tumor region. The firsttarget region can be delineated on the pancreatic phase by a radiologistwith 18 years of experience in manual pancreatic imaging. The secondtarget region can be segmented automatically by a nnUNet (no-new-Net)model trained on a public pancreatic cancer dataset with annotations.The nnUNet model is known in prior art, and a process of segmenting thesecond target region using the nnUNet model is also prior art. Thepresent disclosure will not describe the same in detail herein.

In some embodiments, the array can be prepared for every tumor volumeand the array X_(t)={X_(t) ^(CT), X_(t) ^(M) ^(T) , X_(t) ^(M) ^(P) },t∈{1,2,3}, respectively represent the non-contrast phase, the venousphase, and the pancreatic phase. C_(T) represents an enhanced medicalimage corresponding to the non-contrast phase, M_(T) represents a secondtarget image (tumor mask) corresponding to the venous phase, and M_(P)represents a first target image (pancreas mask) corresponding to thepancreatic phase.

In some embodiments, a pre-trained 3D convolutional neural network modelcan be used to detect a phase of each enhanced medical image in theseries of enhanced medical images. The phase of each enhanced medicalimage in the series of enhanced medical images is re-marked by thepre-trained 3D convolutional neural network model to obtain an accuratephase, thereby enabling effective management of the enhanced medicalimages. Each of the enhanced medical images corresponds to one phase,and different enhanced medical images may correspond to differentphases.

In some embodiments, in order to remove noise from the each enhancedmedical image, the method also includes, before constructing of the dataset, defining a first threshold value and a second threshold valuegreater than the first threshold value; comparing each pixel value inthe enhanced medical image with the first threshold value and comparingeach pixel value in the enhanced medical image with the second thresholdvalue; updating a pixel value according to the first threshold value,and when the pixel value in the enhanced medical image is smaller thanthe first threshold value; updating a pixel value according to thesecond threshold value. When the pixel value in the enhanced medicalimage is greater than the second threshold value, keeping a pixel valueunchanged; and when the pixel value in the enhanced medical image isgreater than the first threshold but less than the second threshold;updating the enhanced medical image according to the updated pixelvalue.

In some embodiments, after updating each enhanced medical image, themethod also includes: obtaining a plurality of resampled enhancedmedical images by resampling each enhanced medical image into anisotropic enhanced medical image; and enhancing the plurality ofresampled enhanced medical images.

All the enhanced medical images can be resampled to an isotropic 1mm{circumflex over ( )}3 resolution. In order to expand the constructeddata set, all of the resampled enhanced medical images in the pluralitycan be enhanced. Training a multi-task prediction model based on theexpanded data set can improve a generalization ability of the multi-taskprediction model.

In some embodiments, the method of enhancing the plurality of resampledenhanced medical images includes: rotating the plurality of resampledenhanced medical images according to a pre-rotation angle; or randomlyzooming the plurality of resampled enhanced medical images.

For example, the pre-rotation angle can be 90°. Rotating the volume oftumors axially around the tumor center with the step size of 90° to getthe corresponding 3D CT image patches and their mirrored patches. Amulti-phase sequence of image sub volumes of 64×64×64 centered at thetumor 3D centroid are cropped to cover the entire tumor and itssurrounding pancreas regions.

In block 12, for each training data set, inputting the training data setinto a first network structure and a second network structure fortraining, extracting first feature maps of the training data set throughthe first network structure, and extracting second feature maps of thetraining data set through the second network structure.

The preoperative multi-phase CE-CT pancreatic imaging used in thepresent disclosure are the result of being scanned at three time pointsfor PDACs located at the pancreas head and uncinate. After thenon-contrast phase, average imaging time delays are 40-50 seconds forthe pancreatic phase and 65-70 seconds for the portal venous phase. FIG.2 shows three examples illustrating different tumor attenuations andresection margins of PDAC patients. Tumor attenuation in specific CTphases is very important characteristic to identify and detect thetumor. Each row in FIG. 2 represents one PDAC patient, and blackboundaries represent the tumor annotations. The white arrow indicates atypical hypo-attenuating tumor, while gray arrow shows aniso-attenuating tumor. In previous studies, Kim et al. stated thatvisually iso-attenuating PDACs are associated with better survival ratesafter surgery, as opposed to typical hypo-attenuating. Hypo-attenuatingmass can be clearly observed in both pancreatic and venous phases of thefirst and second patients, indicating low stromal fractions (worseclinical outcomes). For the third patient, even though tumorhypo-attenuating is observed in pancreatic phase, it then reflects iso-or even hyper-attenuating in the venous phase compared with its adjacentpancreas regions, indicating high stromal fractions (better survival).Tumor enhancement changes across phases is a very useful marker toreflect tumor internal variations and can benefit prognosis.

Besides tumor attenuation, another very important factor is theresection margin status indicating whether cancer cells are presentwithin 1 mm of all resection margins. More specifically, the resectionmargin status is characterized as R0 when no evidence of malignantglands is identified at any of the resection margins. R1 resections havemalignant glands infiltrating at least one of the resection margins onthe permanent section and are usually associated with poor overallsurvival. From the FIG. 2 , both tumors from the first and secondpatient display hypo-attenuating appearances, but clearly the secondtumor has infiltrated out of the pancreas shown in tumor and pancreasmasks. The pathological evidence indicates that the second patient hasthe PDAC with R1 resections, and a follow-up study shows this patienthas worse outcome than the first patient. Radiological observationsabout tumor attenuation and surgical margins status from CE-CT imagingmotivate the development of a preoperative PDAC survival model.

Referring to FIG. 3 , a schematic structural diagram of a networkarchitecture of a multi-task prediction model according to the presentdisclosure is shown. Time points 1, 2, 3 are used to representnon-contrast. The network architecture of a multi-task prediction modelhas two branches: a first branch and a second branch, the first branchused for predicting a resection margin prediction value and the secondbranch used for predicting a survival risk prediction value.

The first branch uses one 3D-CNN model with six convolutional layersequipped with Batch Normalization and ReLu. Input of the first branch isa concatenation of CT volumes at different time points and thecorresponding first and second images: e.g., X∈R^(5×64) ³ . This branchwill try to learn the CT intensity attenuation variations and therelationships between tumor and surrounding pancreas regions, which helpto classify the tumor into different resection statuses. Note that R0/R1can only be obtained after the surgery and pathology. But the multi-taskprediction model can be applied preoperatively in real scenarios tooffer appropriate advice regarding surgical decisions to PDAC patients.

The second branch uses CT volumes at each phase (each phase isCT-M_(T)-M_(P)three-channel input, X_(t)∈R^(3×64) ³ ). An aim of thesecond branch is to capture the tumor attenuation patterns acrossphases. Tumor attenuation usually means the contrast differences betweenthe tumor and its surrounding pancreas regions so both the tumor andpancreas masks are introduced into input volumes. The core part of thisbranch is a recurrence module that allows the network to retain what ithas seen and to update the memory when it sees a new phase image. Anaive approach is to use a vanilla LSTM or ConvLSTM network.Conventional ConvLSTM is capable of modeling 2D spatio-temporal imagesequences by explicitly encoding the 2D spatial structures into thetemporal domain. Contrast-Enhanced 3D Convolutional LSTM (CE-ConvLSTM)network can be used to capture the temporally enhanced patterns fromCE-CT sequences. CE-ConvLSTM can model 4D spatio-temporal CE-CTsequences by explicitly encoding their 3D spatial structures into thetemporal domain. The main equations of ConvLSTM are as follows:

f_(t) = σ(W_(f)^(X) * X_(t)  + W_(f)^(H) * H_(t − 1) + b_(f)), i_(t) = σ(W_(i)^(X) * X_(t) + W_(i)^(H) * H_(t − 1) + b_(i)), o_(t) = σ(W_(o)^(X) * X_(t) + W_(o)^(H) * H_(t − 1) + b_(o)), C_(t) = f_(t) ⊙ C_(t − 1) + i_(t) ⊙ tanh (W_(C)^(X) * X_(t) + W_(C)^(H) * H_(t − 1) + b_(C)), H_(t) = o_(t) ⊙ tanh (C_(t)).

where X_(t) is the CE-CT sequences at time t, * denotes the convolutionoperation, and ⊙ denotes the Hadamard product. All the gates f, i, o,memory cell C, and hidden state H are 4D tensors. 3×3×3 convolutionalkernel and 128 can be used as the channel dimension of hidden states forthe LSTM unit. 3D-ResNet18 can be used as the encoder to encode eachthree-channel input to the lower-dimensional feature maps forCE-ConvLSTM.

In some embodiments, the cropped regions with random shifts can berandomly selected for each iteration during the training process. Thisdata augmentation can improve the network's ability to locate thedesired translational invariants. The batch sizes can be 8. The maximumiteration is set to be 500 epochs.

In block 13, obtaining joint feature maps by connecting the firstfeature maps and the second feature maps, obtaining a resection marginrisk loss value by calculating a resection margin risk loss functionbased on the joint feature maps, and obtaining a survival risk lossvalue by calculating a survival risk loss function based on the jointfeature maps.

In some embodiments, the resection margin risk loss function can be abinary cross-entropy loss function, and the survival risk loss functioncan be L(y_(i))=Σ_(i)δ_(i)(−y_(i)+log Σ_(j:t) _(j) _(≥t) _(i)exp(y_(j))), where j is from the set which has survival time equal to orlarger than t_(i) (t_(j)≥t_(i)).

After the concatenation of the first feature maps and the second featuremaps from both tasks, the channel number of this common representationis 256. Then two separate fully-connected networks will use the commonrepresentation for each prediction task. In the training phase, labelsof the resection status and patient overall survival information (OStime and censoring status) are known for each input of CE-CT sequence.The weighted binary cross-entropy (BCE) loss is applied to the resectionmargin prediction task, while the negative log partial likelihood isused to predict the survival outcomes of a certain patient.

In block 14, determining whether the resection margin risk loss valueand the survival risk loss value meet their respective loss thresholds.

A first risk loss threshold and a second risk loss threshold can bepreset. The first risk loss threshold corresponds to the resectionmargin risk loss value. The second risk loss threshold corresponds tothe survival risk loss value. When the resection margin risk loss valueis less than or equal to the first risk loss threshold, it can bedetermined that the resection margin risk loss value meets the lossthreshold. When the resection margin risk loss value is less than orequal to the second risk loss threshold, it can be determined that theresection margin risk loss value meets the loss threshold.

In block 15, when the resection margin risk loss value and the survivalrisk loss value both meet their respective loss thresholds, stopping thetraining of the first network structure and the second networkstructure, to obtain a plurality of multi-task prediction models.

when both the resection margin risk loss value and the survival riskloss value fail to meet their respective loss thresholds, a plurality oftraining data sets is reacquired from the constructed data set, aplurality of multi-task prediction models being retained.

In block 16, selecting a target multi-task prediction model from theplurality of multi-task prediction models.

However many training data sets there are, there is a one-to-onecorrespondence between training data sets and multi-task predictionmodels.

In some embodiments, the method of selecting a target multi-taskprediction model from the plurality of multi-task prediction modelsincludes: obtaining a plurality of testing data sets from theconstructed data set, each testing data set corresponding to eachtraining data set; obtaining a plurality of testing values by using eachtesting data set to test the multi-task prediction model; determining alargest testing value among the plurality of testing values; anddetermining a multi-task prediction model corresponding to the largesttesting value as the target multi-task prediction model.

A plurality of arrays are randomly obtained from the constructed dataset each time to construct a training data set, and the remaining arraysin the data set can be used to construct a test data set. One test setcan be used to test the corresponding multi-task prediction model toobtain the predicted value. The prediction value is used to indicate aprediction performance of the multi-task prediction model. The largerthe prediction value, the better the prediction performance of themulti-task prediction model; otherwise, the smaller the predictionvalue, the worse the prediction performance of the multi-task predictionmodel. A target multi-task prediction model can be selected from theplurality of multi-task prediction models according to the testingvalues.

In some embodiments, the method of obtaining a plurality of testingvalues by using each testing data set to test the correspondingmulti-task prediction model includes: calculating a mean value and avariance value of each training data set; standardizing each testingdata set according to the mean value and the variance value of thecorresponding testing data set; and obtaining the plurality of testingvalues by using each standardized testing data set to test thecorresponding multi-task prediction model. By normalizing each enhancedmedical image, a training efficiency of a joint learning model can beimproved.

In block 17, obtaining a resection margin prediction value and asurvival risk prediction value by predicting an enhanced medical imageto be measured through the target multi-task prediction model.

The enhanced medical image to be measured can be an enhanced CT imageacquired by using the Computed Tomography to scan PDAC patientspreparing for surgery.

To demonstrate the added value of the proposed signature to the currentstaging system, Kaplan-Meier survival curves are plotted, as in FIG. 4 ,for patients with further stratification by the signature after groupingby TNM and Tumor sizes respectively, these being two well-establishedstratification criteria. The cut-off for radiomics signature is themedian score of each signature. Two subgroups of patients arestudied: 1) patients divided by TNM staging I vs. II-III, and 2)patients divided by the primary PDAC tumor size≤20 mm vs. >20 mm. It isshown that the proposed signature remains the most significant log-ranktest outcome in the subgroup of patients, while the radiomics signaturedoes not reach the statistical significance within the patientsub-population of PDAC tumor≤20 mm. Results shown in FIG. 3 demonstratethat after using the current clinicopathologic TNM staging system ortumor size, the proposed multi-phase CT imaging-based signature canindeed further provide the risk stratification with significantevidence. This novel deep signature can be combined with the establishedclinicopathological criteria to refine the risk stratification and guidethe individualized treatment of PDAC patients.

The enhanced medical images processing device according to the presentdisclosure proposes a multi-task prediction model including a novel 3DContrast-Enhanced Convolutional Long Short-Term Memory (CE-ConvLSTM)network to learn the enhancement dynamics of tumor attenuation frommulti-phase CE-CT images. The multi-task prediction model can capturethe tumor's temporal changes across several phases more effectively thanthe early fusion of input images. Furthermore, to allow the tumorresection margin information to contribute to the survival predictionpreoperatively, the multi-task prediction model can be used to conduct ajoint prediction of a resection margin prediction value and a survivalrisk prediction value. The joint learning of tumor significance andtumor attenuation in a multi-task setting can benefit both tasks andderive more effective/comprehensive prognosis-related deep imagefeatures. Extensive experimental results verify the effectiveness of thepresented framework. The signature built from the proposed model remainsstrong in multivariable analysis adjusting for establishing clinicalpredictors and can be combined with the established criteria for riskstratification and management of PDAC patients.

FIG. 5 shows a schematic structural diagram of an embodiment of apreoperative survival prediction device based on enhanced medical imagesaccording to the present disclosure.

In some embodiments, the preoperative survival prediction device 50 caninclude a plurality of function modules consisting of program codesegments. The program code of each program code segments in the devicefor the preoperative survival prediction device 50 may be stored in amemory of a computing device and executed by the at least one processorto perform (described in detail in FIG. 1 ) a function of preoperativesurvival prediction based on enhanced medical images.

In an embodiment, the preoperative survival prediction device 50 can bedivided into a plurality of functional modules, according to theperformed functions. The functional module can include: a constructionmodule 501, an extraction module 502, an acquisition module 503, adetermination module 504, a training module 505, a selection module 506,and a prediction module 507. A module as referred to in the presentdisclosure refers to a series of computer program segments that can beexecuted by at least one processor and that are capable of performingfixed functions, which are stored in a memory. In this embodiment, thefunctions of each module will be detailed in the following embodiments.

The construction module 501 is configured to construct a data setaccording to a plurality of enhanced medical images and a resectionmargin of each enhanced medical image, and obtain a plurality oftraining data sets from the constructed data set.

In some embodiments, a series of medical images are obtained by scanninga patient's body or a part with an image scanning device afterintravenous injection of contrast media. Medical images obtained byscanning a patient using an image scanning device after intravenousinjection of contrast media are called enhanced medical images. Enhancedmedical image corresponding to different phases between a plurality ofplanes can be considered as one frame in the series of enhanced medicalimages. That is, the series of enhanced medical images will include aplurality of enhanced medical images. With application of contrast mediafor clinical purposes, effectiveness of images is enhanced, and accuracyof diagnoses is improved.

The patient may be a patient suffering from any tumor, such as apancreatic ductal adenocarcinoma, a liver tumor, a lung tumor, alymphoma tumor, a hepatic hemangioma, or the like. The image scanningdevice can utilize, for example, a Computer Tomography (CT), a MagneticResonance Imaging (MRI), a positron emission tomography (PET), a singlephoton emission computed tomography (SPECT), an ultrasound scanning, arotational angiography, and other medical imaging modalities.Correspondingly, the enhanced medical image can be an enhanced. CTimage, an enhanced MRI image, an enhanced PET image, an enhanced SPECTimage, an enhanced ultrasound scan image, and an enhanced rotationalangiography image and other enhanced medical imaging modal images.

In order to facilitate understanding of the present disclosure, thefollowing describes an example of enhanced CT images of PDAC patients.

In some embodiments, the construction module 501 constructing a data setaccording to a plurality of enhanced medical images and a resectionmargin of each enhanced medical image includes: obtaining a plurality offirst target images by delineating a first target region in eachenhanced medical image corresponding to a first phase; obtaining aplurality of second target images by segmenting a second target regionin each enhanced medical image corresponding to a second phase;constructing an array by combining one enhanced medical image and thecorresponding first target image, and the corresponding second targetimage, and the corresponding resection margin, the data set including aplurality of arrays.

Phases shown in the enhanced CT images of the pancreatic ductaladenocarcinoma patient include a non-contrast phase, a pancreatic phase,and a venous phase. The non-contrast phase is before any contrast mediais injected into body of the patient. The pancreatic phase (also calledarterial phase) is when a contrast media is moved from the heart intothe arteries, all structures/organs that get their blood supply from thearteries will show optimal enhancement. In the aorta, a majorenhancement can be observed. The venous phase is when the contrast mediais in the veins, flowing back to the heart. In the venous phase, theliver parenchyma is enhanced through blood supply by the portal vein andsome enhancement of the hepatic veins can be seen. Pancreatic CT scansof 205 patients (with significant PDACs, meaning tumor size=2.5 cm) wereundertaken preoperatively during non-contrast, pancreatic, and portalvenous phases (i.e., 615 CT volumes). Only 24 out of 205 patients haveR1 resection margins, and the imbalance in a resection margin loss isconsidered.

In this embodiment, the first phase can be the pancreatic phase, and thesecond phase can be the venous phase. The first target region in eachenhanced medical image can be a pancreas region. The second targetregion in each enhanced medical image can be a tumor region. The firsttarget region can be delineated on the pancreatic phase by a radiologistwith 18 years of experience in manual pancreatic imaging. The secondtarget region can be segmented automatically by a nnUNet (no-new-Net)model trained on a public pancreatic cancer dataset with annotations.The nnUNet model is known in prior art, and a process of segmenting thesecond target region using the nnUNet model is also prior art. Thepresent disclosure will not describe the same in detail herein.

In some embodiments, the array can be prepared for every tumor volumeand the array X_(t)={X_(t) ^(CT), X_(t) ^(M) ^(T) , X_(t) ^(M) ^(P) },t∈{1, 2, 3}, respectively represent the non-contrast phase, the venousphase, and the pancreatic phase. C_(T) represents an enhanced medicalimage corresponding to the non-contrast phase, M_(T) represents a secondtarget image (tumor mask) corresponding to the venous phase, and M_(P)represents a first target image (pancreas mask) corresponding to thepancreatic phase.

In some embodiments, a pre-trained 3D convolutional neural network modelcan be used, to detect a phase of each enhanced medical image in theseries of enhanced medical images. The phase of each enhanced medicalimage in the series of enhanced medical images is re-marked by thepre-trained 3D convolutional neural network model to obtain an accuratephase, thereby enabling effective management of the enhanced medicalimages. Each of the enhanced medical images corresponds to one phase,and different enhanced medical images may correspond to differentphases.

In some embodiments, in order to remove noise from the each enhancedmedical image, before constructing the data set, the device 50 alsoincluding: define a first threshold value and a second threshold valuegreater than the first threshold value; compare each pixel value in theenhanced medical image with the first threshold value and comparing eachpixel value in the enhanced medical image with the second thresholdvalue; update a pixel value according to the first threshold value, whenthe pixel value in the enhanced medical image is smaller than the firstthreshold value; update a pixel value according to the second thresholdvalue, when the pixel value in the enhanced medical image is greaterthan the second threshold value; keep a pixel value unchanged, when thepixel value in the enhanced medical image is greater than the firstthreshold but less than the second threshold; update the enhancedmedical image according to the updated pixel value.

In some embodiments, a plurality of resampled enhanced medical imagescan be obtained by resampling each enhanced medical image into anisotropic enhanced medical image after updating each enhanced medicalimage; and the plurality of resampled enhanced medical images can beenhanced.

All the enhanced medical images can be resampled to an isotropic 1mm{circumflex over ( )}3 resolution. In order to expand the constructeddata set, all of the resampled enhanced medical images in the pluralitycan be enhanced. Training a multi-task prediction model based on theexpanded data set improves a generalization ability of the multi-taskprediction model.

In some embodiments, by rotating the plurality of resampled enhancedmedical images according to a pre-rotation angle; or randomly zoomingthe plurality of resampled enhanced medical images, the plurality ofresampled enhanced medical images can be enhanced.

For example, the pre-rotation angle can be 90°. Rotating the volume oftumors axially around the tumor center with the step size of 90° to getthe corresponding 3D CT image patches and their mirrored patches. Amulti-phase sequence of image sub volumes of 64×64×64 centered at thetumor 3D centroid are cropped to cover the entire tumor and itssurrounding pancreas regions.

The extraction module 502 is configured to, for each training data set,input the training data set into a first network structure and a secondnetwork structure for training, extract first feature maps of thetraining data set through the first network structure, and extractsecond feature maps of the training data set through the second networkstructure.

The preoperative multi-phase CE-CT pancreatic imaging used in thepresent disclosure are the result of being scanned at three time pointsfor PDACs located at the pancreas head and uncinate. After thenon-contrast phase, average imaging time delays are 40-50 seconds forthe pancreatic phase and 65-70 seconds for the portal venous phase. FIG.2 shows three examples illustrating different tumor attenuations andresection margins of PDAC patients. Tumor attenuation in specific CTphases is very important characteristic to identify and detect thetumor. Each row in FIG. 2 represents one PDAC patient, and blackboundaries represent the tumor annotations. The white arrow indicates atypical hypo-attenuating tumor, while gray arrow shows aniso-attenuating tumor. In previous studies, Kim et al. stated thatvisually iso-attenuating PDACs are associated with better survival ratesafter surgery, as opposed to typical hypo-attenuating. Hypo-attenuatingmass can be clearly observed in both pancreatic and venous phases of thefirst and second patients, indicating low stromal fractions (worseclinical outcomes). For the third patient, even though tumorhypo-attenuating is observed in pancreatic phase, it then reflects iso-or even hyper-attenuating in the venous phase compared with its adjacentpancreas regions, indicating high stromal fractions (better survival).Tumor enhancement changes across phases is a very useful marker toreflect tumor internal variations and can benefit prognosis.

Besides tumor attenuation, another very important factor is theresection margin status indicating whether cancer cells are presentwithin 1 mm of all resection margins. More specifically, the resectionmargin status is characterized as R0 when no evidence of malignantglands is identified at any of the resection margins. R1 resections havemalignant glands infiltrating at least one of the resection margins onthe permanent section and are usually associated with poor overallsurvival. From the FIG. 2 , both tumors from the first and secondpatient display hypo-attenuating appearances, but clearly the secondtumor has infiltrated out of the pancreas shown in tumor and pancreasmasks. The pathological evidence indicates that the second patient hasthe PDAC with R1 resections, and a follow-up study shows this patienthas worse outcome than the first patient. Radiological observationsabout tumor attenuation and surgical margins status from CE-CT imagingmotivate the development of a preoperative PDAC survival model.

Referring to FIG. 3 , a schematic structural diagram of a networkarchitecture of a multi-task prediction model according to the presentdisclosure is shown. Time points 1, 2, 3 are used to representnon-contrast. The network architecture of a multi-task prediction modelhas two branches: a first branch and a second branch, the first branchused for predicting a resection margin prediction value and the secondbranch used for predicting a survival risk prediction value.

The first branch uses one 3D-CNN model with six convolutional layersequipped with Batch Normalization and ReLu. Input of the first branch isa concatenation of CT volumes at different time points and thecorresponding first and second images: e.g., X∈R^(5×64) ³ . This branchwill try to learn the CT intensity attenuation variations and therelationships between tumor and surrounding pancreas regions, which helpto classify the tumor into different resection statuses. Note that R0/R1can only be obtained after the surgery and pathology. But the multi-taskprediction model can be applied preoperatively in real scenarios tooffer appropriate advice regarding surgical decisions to PDAC patients.

The second branch uses CT volumes at each phase (each phase isCT-M_(T)-M_(p)three-channel input, X_(t)∈R^(3×64) ³ ). An aim of thesecond branch is to capture the tumor attenuation patterns acrossphases. Tumor attenuation usually means the contrast differences betweenthe tumor and its surrounding pancreas regions so both the tumor andpancreas masks are introduced into input volumes. The core part of thisbranch is a recurrence module that allows the network to retain what ithas seen and to update the memory when it sees a new phase image. Anaive approach is to use a vanilla LSTM or ConvLSTM network.Conventional ConvLSTM is capable of modeling 2D spatio-temporal imagesequences by explicitly encoding the 2D spatial structures into thetemporal domain. Contrast-Enhanced 3D Convolutional LSTM (CE-ConvLSTM)network can be used to capture the temporally enhanced patterns fromCE-CT sequences. CE-ConvLSTM can model 4D spatio-temporal CE-CTsequences by explicitly encoding their 3D spatial structures into thetemporal domain. The main equations of ConvLSTM are as follows:

f_(t) = σ(W_(f)^(X) * X_(t)  + W_(f)^(H) * H_(t − 1) + b_(f)), i_(t) = σ(W_(i)^(X) * X_(t) + W_(i)^(H) * H_(t − 1) + b_(i)), o_(t) = σ(W_(o)^(X) * X_(t) + W_(o)^(H) * H_(t − 1) + b_(o)), C_(t) = f_(t) ⊙ C_(t − 1) + i_(t) ⊙ tanh (W_(C)^(X) * X_(t) + W_(C)^(H) * H_(t − 1) + b_(C)), H_(t) = o_(t) ⊙ tanh (C_(t)).

where X_(t) is the CE-CT sequences at time t, * denotes the convolutionoperation, and ⊙ denotes the Hadamard product. All the gates f, i, o,memory cell C, and hidden state H are 4D tensors. 3×3×3 convolutionalkernel and 128 can be used as the channel dimension of hidden states forthe LSTM unit. 3D-ResNet18 can be used as the encoder to encode eachthree-channel input to the lower-dimensional feature maps forCE-ConvLSTM.

In some embodiments, the cropped regions with random shifts can berandomly selected for each iteration during the training process. Thisdata augmentation can improve the network's ability to locate thedesired translational invariants. The batch sizes can be 8. The maximumiteration is set to be 500 epochs.

The acquisition module 503 is configured to obtain joint feature maps byconnecting the first feature maps and the second feature maps, obtain aresection margin risk loss value by calculating a resection margin riskloss function based on the joint feature maps, and obtain a survivalrisk loss value by calculating a survival risk loss function based onthe joint feature maps.

In some embodiments, the resection margin risk loss function can be abinary cross-entropy loss function, and the survival risk loss functioncan be L(y_(i))=Σ_(i)δ_(i)(−y_(i)+log Σ_(j:t) _(j) _(≥t) _(i)exp(y_(j))), where j is from the set which has survival time equal to orlarger than t_(i) (t_(j)≥t_(i)).

After the concatenation of the first feature maps and the second featuremaps from both tasks, the channel number of this common representationis 256. Then two separate fully-connected networks will use the commonrepresentation for each prediction task. In the training phase, labelsof the resection status and patient overall survival information (OStime and censoring status) are known for each input of CE-CT sequence.The weighted binary cross-entropy (BCE) loss is applied to the resectionmargin prediction task, while the negative log partial likelihood isused to predict the survival outcomes of a certain patient.

The determination module 504 is configured to determine whether theresection margin risk loss value and the survival risk loss value meettheir respective loss thresholds.

A first risk loss threshold and a second risk loss threshold can bepreset. The first risk loss threshold corresponds to the resectionmargin risk loss value. The second risk loss threshold corresponds tothe survival risk loss value. When the resection margin risk loss valueis less than or equal to the first risk loss threshold, it can bedetermined that the resection margin risk loss value meets the lossthreshold. When the resection margin risk loss value is less than orequal to the second risk loss threshold, it can be determined that theresection margin risk loss value meets the loss threshold.

The training module 505, when the resection margin risk loss value andthe survival risk loss value both meet their respective loss thresholds,is configured to stop the training of the first network structure andthe second network structure to obtain a plurality of multi-taskprediction models.

when both the resection margin risk loss value and the survival riskloss value fail to meet their respective loss thresholds, a plurality oftraining data sets is reacquired from the constructed data set, aplurality of multi-task prediction models being retained.

The selection module 506 is configured to select a target multi-taskprediction model from the plurality of multi-task prediction models.

However many training data sets there are, there is a one-to-onecorrespondence between training data sets and multi-task predictionmodels.

In some embodiments, the selection module 506 selecting a targetmulti-task prediction model from the plurality of multi-task predictionmodels includes: obtaining a plurality of testing data sets from theconstructed data set, each testing data set corresponding to eachtraining data set; obtaining a plurality of testing values by using eachtesting data set to test the multi-task prediction model; determining alargest testing value among the plurality of testing values; anddetermining a multi-task prediction model corresponding to the largesttesting value as the target multi-task prediction model.

A plurality of arrays are randomly obtained from the constructed dataset each time to construct a training data set, and the remaining arraysin the data set can be used to construct a test data set. One test setcan be used to test the corresponding multi-task prediction model toobtain the predicted value. The prediction value is used to indicate aprediction performance of the multi-task prediction model. The largerthe prediction value, the better the prediction performance of themulti-task prediction model; otherwise, the smaller the predictionvalue, the worse the prediction performance of the multi-task predictionmodel. A target multi-task prediction model can be selected from theplurality of multi-task prediction models according to the testingvalues.

In some embodiments, the selection module 506 obtaining a plurality oftesting values by using each testing data set to test the correspondingmulti-task prediction model includes: calculating a mean value and avariance value of each training data set; standardizing each testingdata set according to the mean value and the variance value of thecorresponding testing data set; and obtaining the plurality of testingvalues by using each standardized testing data set to test thecorresponding multi-task prediction model. By normalizing each enhancedmedical image, a training efficiency of a joint learning model can beimproved.

The prediction module 507 is configured to obtain a resection marginprediction value and a survival risk prediction value by predicting anenhanced medical image to be measured through the target multi-taskprediction model.

The enhanced medical image to be measured can be an enhanced. CT imageacquired by using the Computed Tomography to scan PDAC patientspreparing for surgery.

To demonstrate the added value of the proposed signature to the currentstaging system, Kaplan-Meier survival curves are plotted, as in FIG. 4 ,for patients with further stratification by the signature after groupingby TNM and Tumor sizes respectively, these being two well-establishedstratification criteria. The cut-off for radiomics signature is themedian score of each signature. Two subgroups of patients arestudied: 1) patients divided by TNM staging I vs. II-III, and 2)patients divided by the primary PDAC tumor size≤20 mm vs. >20 mm. It isshown that the proposed signature remains the most significant log-ranktest outcome in the subgroup of patients, while the radiomics signaturedoes not reach the statistical significance within the patientsub-population of PDAC tumor≤20 mm. Results shown in FIG. 3 demonstratethat after using the current clinicopathologic TNM staging system ortumor size, the proposed multi-phase CT imaging-based signature canindeed further provide the risk stratification with significantevidence. This novel deep signature can be combined with the establishedclinicopathological criteria to refine the risk stratification and guidethe individualized treatment of PDAC patients.

The enhanced medical images processing device according to the presentdisclosure proposes a multi-task prediction model including a novel 3DContrast-Enhanced Convolutional Long Short-Term Memory (CE-ConvLSTM)network to learn the enhancement dynamics of tumor attenuation frommulti-phase CE-CT images. The multi-task prediction model can capturethe tumor's temporal changes across several phases more effectively thanthe early fusion of input images. Furthermore, to allow the tumorresection margin information to contribute to the survival predictionpreoperatively, the multi-task prediction model can be used to conduct ajoint prediction of a resection margin prediction value and a survivalrisk prediction value. The joint learning of tumor significance andtumor attenuation in a multi-task setting can benefit both tasks andderive more effective/comprehensive prognosis-related deep imagefeatures. Extensive experimental results verify the effectiveness of thepresented framework. The signature built from the proposed model remainsstrong in multivariable analysis adjusting for establishing clinicalpredictors and can be combined with the established criteria for riskstratification and management of PDAC patients.

FIG. 6 shows a schematic structural diagram of a computing deviceaccording to an embodiment of the present disclosure.

As shown in FIG. 6 , the computing device 600 may include: at least onestorage device 601, at least one processor 602, at least onecommunication bus 603, and a transceiver 604.

It should be understood by those skilled in the art that the structureof the computing device 600 shown in FIG. 6 does not constitute alimitation of the embodiment of the present disclosure. The computingdevice 600 may be a bus type structure or a star type structure, and thecomputing device 600 may also include more or less hardware or softwarethan illustrated, or may have different component arrangements.

In at least one embodiment, the computing device 600 can include aterminal that is capable of automatically performing numericalcalculations and/or information processing in accordance with pre-set orstored instructions. The hardware of the terminal can include, but isnot limited to, a microprocessor, an application specific integratedcircuit, programmable gate arrays, digital processors, and embeddeddevices. The computing device 600 may further include an electronicdevice. The electronic device can interact with a user through akeyboard, a mouse, a remote controller, a touch panel or a voice controldevice, for example, an individual computers, tablets, smartphones,digital cameras, etc.

It should be noted that the computing device 600 is merely an example,and other existing or future electronic products may be included in thescope of the present disclosure, and are included in the reference.

In some embodiments, the storage device 601 can be used to store programcodes of computer readable programs and various data, such as the devicefor automatically delineating a clinical target volume of esophagealcancer 30 installed in the computing device 600, and automaticallyaccess to the programs or data with high speed during running of thecomputing device 600. The storage device 601 can include a read-onlymemory (ROM), a programmable read-only memory (PROM), an erasableprogrammable read only memory (EPROM), an one-time programmableread-only memory (OTPROM), an electronically-erasable programmableread-only memory (EEPROM), a compact disc read-only memory (CD-ROM), orother optical disk storage, magnetic disk storage, magnetic tapestorage, or any other non-transitory storage medium readable by thecomputing device 600 that can be used to carry or store data.

In some embodiments, the at least one processor 602 may be composed ofan integrated circuit, for example, may be composed of a single packagedintegrated circuit, or may be composed of a plurality of integratedcircuits of same function or different functions. The at least oneprocessor 602 can include one or more central processing units (CPU), amicroprocessor, a digital processing chip, a graphics processor, andvarious control chips. The at least one processor 602 is a control unitof the computing device 600, which connects various components of thecomputing device 600 using various interfaces and lines. By running orexecuting a computer program or modules stored in the storage device601, and by invoking the data stored in the storage device 601, the atleast one processor 602 can perform various functions of the computingdevice 600 and process data of the computing device 600.

In some embodiments, the least one bus 603 is used to achievecommunication between the storage device 601 and the at least oneprocessor 602, and other components of the compute device 600.

Although it is not shown, the computing device 600 may further include apower supply (such as a battery) for powering various components. Insome embodiments, the power supply may be logically connected to the atleast one processor 602 through a power management device, thereby, thepower management device manages functions such as charging, discharging,and power management. The power supply may include one or more a DC orAC power source, a recharging device, a power failure detection circuit,a power converter or inverter, a power status indicator, and the like.The computing device 600 may further include various sensors, such as aBLUETOOTH module, a Wi-Fi module, and the like, and details are notdescribed herein.

It should be understood that the described embodiments are forillustrative purposes only and are not limited by the scope of thepresent disclosure.

The above-described integrated unit implemented in a form of softwarefunction modules can be stored in a computer readable storage medium.The above software function modules are stored in a storage medium, andincludes a plurality of instructions for causing a computing device(which may be a personal computer, or a network device, etc.) or aprocessor to execute the method according to various embodiments of thepresent disclosure.

In a further embodiment, in conjunction with FIG. 1 , the at least oneprocessor 602 can execute an operating device and various types ofapplications (such as the preoperative survival prediction device 50)installed in the computing device 600, program codes, and the like. Forexample, the at least one processor 602 can execute the modules 501-507.

In at least one embodiment, the storage device 601 stores program codes.The at least one processor 602 can invoke the program codes stored inthe storage device 601 to perform related functions. For example, themodules described in FIG. 5 are program codes stored in the storagedevice 601 and executed by the at least one processor 602, to implementthe functions of the various modules.

In at least one embodiment, the storage device 601 stores a plurality ofinstructions that are executed by the at least one processor 602 toimplement all or part of the steps of the method described in theembodiments of the present disclosure.

Specifically, the storage device 601 stores the plurality ofinstructions which when executed by the at least one processor 602,causes the at least one processor 602 to: construct a data setaaccording to a plurality of enhanced medical images and a resectionmargin of each enhanced medical image, and obtain a plurality oftraining data sets from the constructed data set; for each training dataset, input the training data set into a first network structure and asecond network structure for training, extract first feature maps of thetraining data sets through the first network structure, and extractsecond feature maps of the training data sets through the second networkstructure; obtain joint feature maps by connecting the first featuremaps and the second feature maps, obtain a resection margin risk lossvalue by calculating a resection margin risk loss function based on thejoint feature maps, and obtain a survival risk loss value by calculatinga survival risk loss function based on the joint feature maps; determinewhether the resection margin risk loss value and the survival risk lossvalue meet their respective loss thresholds; when the resection marginrisk loss value and the survival risk loss value both meet theirrespective loss thresholds, stop training the first network structureand the second network structure to obtain a plurality of multi-taskprediction models; select a target multi-task prediction model from theplurality of multi-task prediction models; obtain a resection marginprediction value and a survival risk prediction value by predicting anenhanced medical image to be measured through the target multi-taskprediction model.

The embodiment of the present disclosure further provides a computerstorage medium, and the computer storage medium store a program thatperforms all or part of the steps including any of the method describedin the above embodiments.

A non-transitory storage medium having stored thereon instructions that,when executed by a processor of a computing device, causes the computingdevice to perform an preoperative survival prediction method, the methodincluding: constructing a data seta according to a plurality of enhancedmedical images and a resection margin of each enhanced medical image,and obtaining a plurality of training data sets from the constructeddata set; for each training data set, inputting the training data setinto a first network structure and a second network structure fortraining, extracting first feature maps of the training data setsthrough the first network structure, and extracting second feature mapsof the training data sets through the second network structure;obtaining joint feature maps by connecting the first feature maps andthe second feature maps, obtaining a resection margin risk loss value bycalculating a resection margin risk loss function based on the jointfeature maps, and obtaining a survival risk loss value by calculating asurvival risk loss function based on the joint feature maps; determiningwhether the resection margin risk loss value and the survival risk lossvalue meet their respective loss thresholds; when the resection marginrisk loss value and the survival risk loss value both meet theirrespective loss thresholds, stop training the first network structureand the second network structure to obtain a plurality of multi-taskprediction models; selecting a target multi-task prediction model fromthe plurality of multi-task prediction models; obtaining a resectionmargin prediction value and a survival risk prediction value bypredicting an enhanced medical image to be measured through the targetmulti-task prediction model.

It should be noted that, for a simple description, the above methodembodiments expressed as a series of action combinations, but thoseskilled in the art should understand that the present disclosure is notlimited by the described action sequence. According to the presentdisclosure, some steps in the above embodiments can be performed inother sequences or simultaneously. Secondly, those skilled in the artshould also understand that the embodiments described in thespecification are all optional embodiments, and the actions and unitsinvolved are not necessarily required by the present disclosure.

In the above embodiments, descriptions of each embodiment has differentfocuses, and when there is no detail part in a certain embodiment,please refer to relevant parts of other embodiments.

In several embodiments provided in the preset application, it should beunderstood that the disclosed apparatus can be implemented in otherways. For example, the device embodiments described above are merelyillustrative. For example, divisions of the unit are only a logicalfunction division, and there can be other division ways in actualimplementation.

The modules described as separate components may or may not bephysically separated, and the components displayed as modules may or maynot be physical units. That is, it can locate in one place, ordistribute to a plurality of network units. Some or all of the modulescan be selected according to actual needs to achieve the purpose of thesolution of above embodiments.

In addition, each functional unit in each embodiment of the presentdisclosure can be integrated into one processing unit, or can bephysically present separately in each unit, or two or more units can beintegrated into one unit. The above integrated unit can be implementedin a form of hardware or in a form of a software functional unit.

It is apparent to those skilled in the art that the present disclosureis not limited to the details of the above-described exemplaryembodiments, and the present disclosure can be embodied in otherspecific forms without departing from the spirit or essentialcharacteristics of the present disclosure. Therefore, the presentembodiments are to be considered as illustrative and not restrictive,and the scope of the present disclosure is defined by the appendedclaims instead all changes in the meaning and scope of equivalentelements are included in the present disclosure. Any reference signs inthe claims should not be construed as limiting the claim.

The above embodiments are only used to illustrate technical solutions ofthe present disclosure, and are not restrictions on the technicalsolutions. Although the present disclosure has been described in detailwith reference to the above embodiments, those skilled in the art shouldunderstand that the technical solutions described in one embodiments canbe modified, or some of technical features can be equivalentlysubstituted, and these modifications or substitutions do not detractfrom the essence of the technical solutions or from the scope of thetechnical solutions of the embodiments of the present disclosure.

We claim:
 1. A preoperative survival prediction method based on enhancedmedical images applicable in a computing device, the method comprising:constructing a data seta according to a plurality of enhanced medicalimages and a resection margin of each enhanced medical image, andobtaining a plurality of training data sets from the constructed dataset; for each training data set, inputting the training data set into afirst network structure and a second network structure for training,extracting first feature maps of the training data sets through thefirst network structure, and extracting second feature maps of thetraining data sets through the second network structure; obtaining jointfeature maps by connecting the first feature maps and the second featuremaps, obtaining a resection margin risk loss value by calculating aresection margin risk loss function based on the joint feature maps, andobtaining a survival risk loss value by calculating a survival risk lossfunction based on the joint feature maps; determining whether theresection margin risk loss value and the survival risk loss value meettheir respective loss thresholds; when the resection margin risk lossvalue and the survival risk loss value both meet their respective lossthresholds, stopping the training of the first network structure and thesecond network structure, to obtain a plurality of multi-task predictionmodels; selecting a target multi-task prediction model from theplurality of multi-task prediction models; obtaining a resection marginprediction value and a survival risk prediction value by predicting anenhanced medical image to be measured through the target multi-taskprediction model.
 2. The preoperative survival prediction method ofclaim 1, wherein the method of constructing a data seta according to aplurality of enhanced medical images and a resection margin of eachenhanced medical image comprises: obtaining a plurality of first targetimages by delineating a first target region in each enhanced medicalimage corresponding to a first phase; obtaining a plurality of secondtarget images by segmenting a second target region in each enhancedmedical image corresponding to a second phase; constructing an array bycombining one enhanced medical image and the corresponding first targetimage, the corresponding second target image, and the correspondingresection margin, the data set including a plurality of the arrays. 3.The preoperative survival prediction method of claim 2, furthercomprising: defining a first threshold value and a second thresholdvalue greater than the first threshold value; comparing each pixel valuein the enhanced medical image with the first threshold value andcomparing each pixel value in the enhanced medical image with the secondthreshold value; updating a pixel value according to the first thresholdvalue, when the pixel value in the enhanced medical image is smallerthan the first threshold value; updating a pixel value according to thesecond threshold value, when the pixel value in the enhanced medicalimage is greater than the second threshold value; keeping a pixel valueunchanged, when the pixel value in the enhanced medical image is greaterthan the first threshold but less than the second threshold; updatingthe enhanced medical image according to the updated pixel value.
 4. Thepreoperative survival prediction method of claim 3, wherein the methodof selecting a target multi-task prediction model from the plurality ofmulti-task prediction models comprises: obtaining a plurality of testingdata sets from the constructed data set, each testing data setcorresponding to each training data set; obtaining a plurality oftesting values by using each testing data set to test the correspondingmulti-task prediction model; determining a largest testing value amongthe plurality of testing values; and determining a multi-task predictionmodel corresponding to the largest testing value as the targetmulti-task prediction model.
 5. The preoperative survival predictionmethod of claim 4, wherein the method of obtaining a plurality oftesting values by using each testing data set to test the correspondingmulti-task prediction model comprises: calculating a mean value and avariance value of each training data set; standardizing each testingdata set according to the mean value and the variance value of thecorresponding testing data set; and obtaining the plurality of testingvalues by using each standardized testing data set to test thecorresponding multi-task prediction model.
 6. The preoperative survivalprediction method of claim 5, further comprising: obtaining a pluralityof resampled enhanced medical images by resampling each enhanced medicalimage into an isotropic enhanced medical image; and enhancing theplurality of resampled enhanced medical images, comprising rotating theplurality of resampled enhanced medical images according to apre-rotation angle; or randomly zooming the plurality of resampledenhanced medical images.
 7. The preoperative survival prediction methodof claim 5, wherein the first network structure being a 3D-CNN modelwith six convolutional layers equipped with batch normalization andrelu, the second network structure being a CE-ConvLSTM model with a ResTet model cascaded before, the resection margin risk loss function beinga binary cross-entropy loss function, and the survival risk lossfunction being L(y_(i))=Σ_(i)δ_(i)(−y_(i)+log Σ_(j:t) _(j) _(≥t) _(i)exp(y_(j))), where j is from the set whose survival time is equal orlarger than t_(i) (t_(j)≥t_(i)).
 8. A computing device, comprising: atleast one processor; and a storage device storing one or more programswhich when executed by the at least one processor, causes the at leastone processor to: construct a data seta according to a plurality ofenhanced medical images and a resection margin of each enhanced medicalimage, and obtain a plurality of training data sets from the constructeddata set; for each training data set, input the training data set into afirst network structure and a second network structure for training,extract first feature maps of the training data sets through the firstnetwork structure, and extract second feature maps of the training datasets through the second network structure; obtain joint feature maps byconnecting the first feature maps and the second feature maps, obtain aresection margin risk loss value by calculating a resection margin riskloss function based on the joint feature maps, and obtain a survivalrisk loss value by calculating a survival risk loss function based onthe joint feature maps; determine whether the resection margin risk lossvalue and the survival risk loss value meet their respective lossthresholds; when the resection margin risk loss value and the survivalrisk loss value both meet their respective loss thresholds, stop thetraining of the first network structure and the second networkstructure, to obtain a plurality of multi-task prediction models; selecta target multi-task prediction model from the plurality of multi-taskprediction models; obtain a resection margin prediction value and asurvival risk prediction value by predicting an enhanced medical imageto be measured through the target multi-task prediction model.
 9. Thecomputing device of claim 8, wherein the method of constructing a dataseta according to a plurality of enhanced medical images and a resectionmargin of each enhanced medical image comprises: obtaining a pluralityof first target images by delineating a first target region in eachenhanced medical image corresponding to a first phase; obtaining aplurality of second target images by segmenting a second target regionin each enhanced medical image corresponding to a second phase;constructing an array by combining one enhanced medical image and thecorresponding first target image, the corresponding second target image,and the corresponding resection margin, the data set including aplurality of the arrays.
 10. The computing device of claim 9, the atleast one processor further to: define a first threshold value and asecond threshold value greater than the first threshold value; compareeach pixel value in the enhanced medical image with the first thresholdvalue and comparing each pixel value in the enhanced medical image withthe second threshold value; update a pixel value according to the firstthreshold value, when the pixel value in the enhanced medical image issmaller than the first threshold value; update a pixel value accordingto the second threshold value, when the pixel value in the enhancedmedical image is greater than the second threshold value; keep a pixelvalue unchanged, when the pixel value in the enhanced medical image isgreater than the first threshold but less than the second threshold;update the enhanced medical image according to the updated pixel value.11. The computing device of claim 10, wherein the method of selecting atarget multi-task prediction model from the plurality of multi-taskprediction models comprises: obtaining a plurality of testing data setsfrom the constructed data set, each testing data set corresponding toeach training data set; obtaining a plurality of testing values by usingeach testing data set to test the corresponding multi-task predictionmodel; determining a largest testing value among the plurality oftesting values; and determining a multi-task prediction modelcorresponding to the largest testing value as the target multi-taskprediction model.
 12. The computing device of claim 11, wherein themethod of obtaining a plurality of testing values by using each testingdata set to test the corresponding multi-task prediction modelcomprises: calculating a mean value and a variance value of eachtraining data set; standardizing each testing data set according to themean value and the variance value of the corresponding testing data set;and obtaining the plurality of testing values by using each standardizedtesting data set to test the corresponding multi-task prediction model.13. The computing device of claim 12, the at least one processor furtherto: obtain a plurality of resampled enhanced medical images byresampling each enhanced medical image into an isotropic enhancedmedical image; and enhance the plurality of resampled enhanced medicalimages, comprising rotating the plurality of resampled enhanced medicalimages according to a pre-rotation angle; or randomly zooming theplurality of resampled enhanced medical images.
 14. The computing deviceof claim 12, wherein the first network structure being a 3D-CNN modelwith six convolutional layers equipped with batch normalization andrelu, the second network structure being a CE-ConvLSTM model with aResNet model cascaded before, the resection margin risk loss functionbeing a binary cross-entropy loss function, and the survival risk lossfunction being L(y_(i))=Σ_(i)δ_(i)(−y_(i)+log Σ_(j:t) _(j) _(≥t) _(i)exp(y_(j))), where j is from the set whose survival time is equal orlarger than t_(i) (t_(j)≥t_(i)).
 15. A non-transitory storage mediumhaving stored thereon instructions that, when executed by a processor ofa computing device, causes the computing device to perform apreoperative survival prediction method based on enhanced medicalimages, the method comprising: constructing a data seta according to aplurality of enhanced medical images and a resection margin of eachenhanced medical image, and obtaining a plurality of training data setsfrom the constructed data set; for each training data set, inputting thetraining data set into a first network structure and a second networkstructure for training, extracting first feature maps of the trainingdata sets through the first network structure, and extracting secondfeature maps of the training data sets through the second networkstructure; obtaining joint feature maps by connecting the first featuremaps and the second feature maps, obtaining a resection margin risk lossvalue by calculating a resection margin risk loss function based on thejoint feature maps, and obtaining a survival risk loss value bycalculating a survival risk loss function based on the joint featuremaps; determining whether the resection margin risk loss value and thesurvival risk loss value meet their respective loss thresholds; when theresection margin risk loss value and the survival risk loss value bothmeet their respective loss thresholds, stopping the training of thefirst network structure and the second network structure to obtain aplurality of multi-task prediction models; selecting a target multi-taskprediction model from the plurality of multi-task prediction models;obtaining a resection margin prediction value and a survival riskprediction value by predicting an enhanced medical image to be measuredthrough the target multi-task prediction model.
 16. The non-transitorystorage medium of claim 15, wherein the method of constructing a dataseta according to a plurality of enhanced medical images and a resectionmargin of each enhanced medical image comprises: obtaining a pluralityof first target images by delineating a first target region in eachenhanced medical image corresponding to a first phase; obtaining aplurality of second target images by segmenting a second target regionin each enhanced medical image corresponding to a second phase;constructing an array by combining one enhanced medical image and thecorresponding first target image, the corresponding second target image,and the corresponding resection margin, the data set including aplurality of the arrays.
 17. The non-transitory storage medium of claim16, further comprising: defining a first threshold value and a secondthreshold value greater than the first threshold value; comparing eachpixel value in the enhanced medical image with the first threshold valueand comparing each pixel value in the enhanced medical image with thesecond threshold value; updating a pixel value according to the firstthreshold value, when the pixel value in the enhanced medical image issmaller than the first threshold value; updating a pixel value accordingto the second threshold value, when the pixel value in the enhancedmedical image is greater than the second threshold value; keeping apixel value unchanged, when the pixel value in the enhanced medicalimage is greater than the first threshold but less than the secondthreshold; updating the enhanced medical image according to the updatedpixel value.
 18. The non-transitory storage medium of claim 17, whereinthe method of selecting a target multi-task prediction model from theplurality of multi-task prediction models comprises: obtaining aplurality of testing data sets from the constructed data set, eachtesting data set corresponding to each training data set; obtaining aplurality of testing values by using each testing data set to test thecorresponding multi-task prediction model; determining a largest testingvalue among the plurality of testing values; and determining amulti-task prediction model corresponding to the largest testing valueas the target multi-task prediction model.
 19. The non-transitorystorage medium of claim 18, wherein the method of obtaining a pluralityof testing values by using each testing data set to test thecorresponding multi-task prediction model comprises: calculating a meanvalue and a variance value of each training data set; standardizing eachtesting data set according to the mean value and the variance value ofthe corresponding testing data set; and obtaining the plurality oftesting values by using each standardized testing data set to test thecorresponding multi-task prediction model.
 20. The non-transitorystorage medium of claim 18, further comprising: obtaining a plurality ofresampled enhanced medical images by resampling each enhanced medicalimage into an isotropic enhanced medical image; and enhancing theplurality of resampled enhanced medical images, comprising rotating theplurality of resampled enhanced medical images according to apre-rotation angle; or randomly zooming the plurality of resampledenhanced medical images.